Milestone 2: Model Validation

tl;dr

This notebook determines the model framework utilized for this PRD. It utilizes the results from the hyperparameter tuning step, and trains the optimal model for each algorithm on the v67 dataset. The resultant predictions of these optimal models are compared to a v68 validation dataset. The framework used to generate the highest performant model (e.g., features, hyperparameters) will be used in online model training for the subsetting of Beta in production use.

Features

Feature sets:

1. Only performance metrics.
2. Performance metrics and the highest found by Boruta with original dataset.
3. Performance metrics and the highest found by Boruta with equal labels dataset.
4. Performance metrics and all covariates.
5. Only the highest covariates found by Boruta (excluding performance).
6. Only the highest covariates found by Boruta (equal labels, excluding performance).
7. Utilize all covariates (excluding performance).

Validation Bootstraps

30 bootstrap replicates of v68 Release will be utilized to validate the models. There are 257697 profiles utilized for this purpose.

[1] "Loading previously created bootstraps from file"

Oversampling

The entire v67 dataset will be utilized for training. However, most of the matching algorithms are more performant with higher ratios of Beta to Release. Therefore, various levels of oversampled datasets are created for training. There are 70345 profiles utilized for this purpose.

[1] "Loading previously created oversampling training sets from file"

Model Training

Two matching methods were ultimately tested, as CEM and subclassing were producing extremely poor results.

• nearest-neighbors
• genetic matching

For all models the following diagnostics are reported:

• mean, median score against the 30 validation replicates
• Number of matched Beta samples
• QQ and ridge plots of original and matched Beta samples

Nearest-Neighbors, Malahanobis

Hyperparameters: hyperparameter_tuning_nn_malahanobis.Rmd

Feature selection and hyperparameter tuning for the nearest-neighbors models were all trained using the MatchIt library. These models are trained using the full, relevantly oversampled, training dataset.

FS 1

• caliper = 0.20
• calcloset = TRUE
• ratio = 2
• replace = FALSE
• 4x oversampling

       mean      median 
0.001665983 0.001671272 

Number of Beta samples: 29814

FS 3

• caliper = 3.00
• calcloset = TRUE
• ratio = 1
• replace = TRUE
• 16x oversampling

       mean      median 
0.004966338 0.004966342 

Number of Beta samples: 3259

FS 5

• caliper = 0
• calcloset = TRUE
• ratio = 1
• replace = FALSE
• 8x oversampling

       mean      median 
0.004599767 0.004598880 

Number of Beta samples: 6208

Nearest-Neighbors Logit, Linear

Hyperparameters: hyperparameter_tuning_nn_logit_linear.Rmd

FS 1

• caliper = 0.4
• calcloset = TRUE
• ratio = 3
• replace = FALSE
• 4x oversampling
• Interactions across covariates

       mean      median 
0.001789146 0.001795468 

Number of Beta samples: 44721

FS 5

• caliper = 0
• calcloset = FALSE
• ratio = 1
• replace = FALSE
• 8x oversampling

       mean      median 
0.005422051 0.005421492 

Nearest-Neighbors GAM, Logit

Hyperparameters: hyperparameter_tuning_nn_gam_logit.Rmd

FS 1

• caliper = 0.5
• calcloset = FALSE
• ratio = 3
• replace = FALSE
• 4x oversampling
• Interactions across covariates

       mean      median 
0.003007450 0.003011005 

Number of Beta samples: 44721

FS 5

• caliper = 0
• calcloset = FALSE
• ratio = 1
• replace = FALSE
• 4x oversampling

       mean      median 
0.006251073 0.006245329 

Number of Beta samples: 44721

Nearest-Neighbors Probit, Linear

Hyperparameters: hyperparameter_tuning_nn_probit_linear.Rmd

FS 1

• caliper = 0.4
• calcloset = FALSE
• ratio = 3
• replace = FALSE
• 4x oversampling
• Interactions across covariates

       mean      median 
0.001773316 0.001777554 

Number of Beta samples: 44721

FS 5

• caliper = 0
• calcloset = FALSE
• ratio = 1
• replace = FALSE
• 8x oversampling

       mean      median 
0.004524079 0.004523233 

Number of Beta samples: 7453

Genetic Matching

Due to the very long training times of genetic matching, feature selection and hyperparameter tuning were not performed using bootstrap sampes. The models were previously trained using the full training sample. The resultant match beta subsets were serialized and uploaded to GCP.

Trained: feature_selection_genmatch_linear.Rmd

Successfully auto-authenticated via moz-fx-dev-cdowhyglund-subBeta-788f8f0d4627.json
Set default bucket name to 'moz-fx-dev-subbeta'
2019-11-21 07:57:28 -- Saved data/milestone2/feature_selection_genmatch_.RData to data/feature_selection_genmatch_.RData (166.4 Mb)

[1] TRUE

FS 1

       mean      median 
0.001500821 0.001506689 

Number of Beta samples: 44985

Pop.Size 500

       mean      median 
0.001547905 0.001553225 

Number of Beta samples: 44982

FS 3

       mean      median 
0.003216196 0.003224829 

Number of Beta samples: 44982

FS 5

       mean      median 
0.003283770 0.003288647 

Number of Beta samples: 44982

Results

Genetic matching produces the optimal matching results. It yields the lowest scores, without dropping large sample of the Beta profiles. Unsurprisingly, matching directly, and only, on the performance metrics yields the best results. Increasing the population size hyperparameter slightly decreased model performance, which is surprising. Considering this substantially increases training times, the lower value will be used.

Conclusion: Optimal model framework

• matching algorithm: Genetic Matching
• features: Only performance metrics
• hyperparameters: Default for Matching library

Push to GCP

The boostrap replicates for validation, oversampled traininig datasets, resultant quantile calculations, and matched datasets are saved to an R image. Then uploaded to the project GCP bucket.

[1] "Previously trained results already exist: data/milestone2/validation_results_20191106.RData"